We study the task of replicating the functionality of black-box neural models, for which we only know the output class probabilities provided for a set of input images. We assume back-propagation through the black-box model is not possible and its training images are not available, e.g. the model could be exposed only through an API. In this context, we present a teacher-student framework that can distill the black-box (teacher) model into a student model with minimal accuracy loss. To generate useful data samples for training the student, our framework (i) learns to generate images on a proxy data set (with images and classes different from those used to train the black-box) and (ii) applies an evolutionary strategy to make sure that each generated data sample exhibits a high response for a specific class when given as input to the black box. Our framework is compared with several baseline and state-of-the-art methods on three benchmark data sets. The empirical evidence indicates that our model is superior to the considered baselines. Although our method does not back-propagate through the black-box network, it generally surpasses state-of-the-art methods that regard the teacher as a glass-box model.

We thank reviewers for their useful comments and insights. We next address concerns raised by the reviewers. The evolutionary algorithm might collapse to certain regions of the output space. This demonstrates that the proposed evolutionary strategy does not collapse to certain regions. Did the authors look at mode collapse of GAN or blurry effect of V AE?

Additional Feedback: The paper can be imporved by including experiments and comparison with baseline on more practical dataset. My main concerns on the dataset and results in many classes have been addressed. Black-box model stealing is also discussed in [1]. Considering previous work[1] in the context, using EA on a pre-trained GAN for model stealing is not so novel. The novelty is limited in using EA in this task.

The basic idea of evolving a data set that mimics a black box model of a teacher is intuitive and interesting, although not entirely novel. But the combination of EA and GAN algorithms for realizing a solution to this problem is. The experimental results presented show superiority over the tested baselines, and the paper is well written and easy to understand. One limitation is that the method only evaluated on small data sets and the description of the EA that is used needs to be better explained. The options of several reviewers were raised as a result of clarifications provided in the user response, and the consensus recommendation on this paper is to accept. Please be sure to attend to the reviewer comments as you prepare your final version.

We study the task of replicating the functionality of black-box neural models, for which we only know the output class probabilities provided for a set of input images. We assume back-propagation through the black-box model is not possible and its training images are not available, e.g. the model could be exposed only through an API. In this context, we present a teacher-student framework that can distill the black-box (teacher) model into a student model with minimal accuracy loss. To generate useful data samples for training the student, our framework (i) learns to generate images on a proxy data set (with images and classes different from those used to train the black-box) and (ii) applies an evolutionary strategy to make sure that each generated data sample exhibits a high response for a specific class when given as input to the black box. Our framework is compared with several baseline and state-of-the-art methods on three benchmark data sets.